notebooks/tutorials/storage/Storage command-line tool.ipynb (328 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Storage command-line tool\n",
"\n",
"The [Google Cloud SDK](https://cloud-dot-devsite.googleplex.com/sdk/docs/) provides a set of commands for working with data stored in Cloud Storage. This notebook introduces several `gsutil` commands for interacting with Cloud Storage. Note that shell commands in a notebook must be prepended with a `!`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## List available commands\n",
"\n",
"The `gsutil` command can be used to perform a wide array of tasks. Run the `help` command to view a list of available commands:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!gsutil help"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create a storage bucket\n",
"\n",
"Buckets are the basic containers that hold your data. Everything that you store in Cloud Storage must be contained in a bucket. You can use buckets to organize your data and control access to your data.\n",
"\n",
"Start by defining a globally unique name.\n",
"\n",
"For more information about naming buckets, see [Bucket name requirements](https://cloud.google.com/storage/docs/naming#requirements)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Replace the string below with a unique name for the new bucket\n",
"bucket_name = \"your-new-bucket\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"NOTE: In the examples below, the `bucket_name` and `project_id` variables are referenced in the commands using `{}` and `$`. If you want to avoid creating and using variables, replace these interpolated variables with literal values and remove the `{}` and `$` characters.\n",
"\n",
"Next, create the new bucket with the `gsutil mb` command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gsutil mb gs://{bucket_name}/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## List buckets in a project\n",
"\n",
"Replace 'your-project-id' in the cell below with your project ID and run the cell to list the storage buckets in your project."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Replace the string below with your project ID\n",
"project_id = \"your-project-id\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gsutil ls -p $project_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response should look like the following:\n",
"\n",
"```\n",
"gs://your-new-bucket/\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get bucket metadata\n",
"\n",
"The next cell shows how to get information on metadata of your Cloud Storage buckets.\n",
"\n",
"To learn more about specific bucket properties, see [Bucket locations](https://cloud.google.com/storage/docs/locations) and [Storage classes](https://cloud.google.com/storage/docs/storage-classes)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!gsutil ls -L -b gs://{bucket_name}/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response should look like the following:\n",
"```\n",
"gs://your-new-bucket/ :\n",
" Storage class: MULTI_REGIONAL\n",
" Location constraint: US\n",
" ...\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Upload a local file to a bucket\n",
"\n",
"Objects are the individual pieces of data that you store in Cloud Storage. Objects are referred to as \"blobs\" in the Python client library. There is no limit on the number of objects that you can create in a bucket.\n",
"\n",
"An object's name is treated as a piece of object metadata in Cloud Storage. Object names can contain any combination of Unicode characters (UTF-8 encoded) and must be less than 1024 bytes in length.\n",
"\n",
"For more information, including how to rename an object, see the [Object name requirements](https://cloud.google.com/storage/docs/naming#objectnames)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gsutil cp resources/us-states.txt gs://{bucket_name}/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## List blobs in a bucket"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!gsutil ls -r gs://{bucket_name}/**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response should look like the following:\n",
"```\n",
"gs://your-new-bucket/us-states.txt\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get a blob and display metadata\n",
"\n",
"See [Viewing and editing object metadata](https://cloud.google.com/storage/docs/viewing-editing-metadata) for more information about object metadata."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gsutil ls -L gs://{bucket_name}/us-states.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response should look like the following:\n",
"\n",
"```\n",
"gs://your-new-bucket/us-states.txt:\n",
" Creation time: Fri, 08 Feb 2019 05:23:28 GMT\n",
" Update time: Fri, 08 Feb 2019 05:23:28 GMT\n",
" Storage class: STANDARD\n",
" Content-Language: en\n",
" Content-Length: 637\n",
" Content-Type: text/plain\n",
"...\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download a blob to a local directory"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!gsutil cp gs://{bucket_name}/us-states.txt resources/downloaded-us-states.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Delete a blob"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!gsutil rm gs://{bucket_name}/us-states.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Delete a bucket\n",
"\n",
"The following command deletes all objects in the bucket before deleting the bucket itself."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gsutil rm -r gs://{bucket_name}/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next Steps\n",
"\n",
"Read more about Cloud Storage in the documentation:\n",
"+ [Storage key terms](https://cloud.google.com/storage/docs/key-terms)\n",
"+ [How-to guides](https://cloud.google.com/storage/docs/how-to)\n",
"+ [Pricing](https://cloud.google.com/storage/pricing)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 1
}